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The Impact of fhe 2012 TNCore Math Training on Teaching Practices and Effectiveness 


EXECUTIVE SUMMARY 


D uring the spring and summer of 2012, the Tennessee Department of Education trained about 200 
Common Core Coaches who then went on to facilitate summer trainings for thousands of the state's 
third through eighth grade math teachers. The following summer, the training sessions reached nearly 
30,000 teachers across the state, covering math, English language arts, and literacy in science, social 
studies, and career and technical education. 

The Tennessee Core Coach training model was designed to develop a network of teachers with a deep 
content and pedagogical knowledge of the Common Core State Standards who could pass the knowledge on 
to their peers during formal training sessions and informal interactions throughout the year. Coaches were 
Tennessee teachers selected via a competitive application and interview process. Coaches received eight 
days of intensive grade-level training provided by the Institute for Learning at the University of Pittsburgh, 
engaging with the material first as learners and then as teacher trainers. Coaches then delivered training to 
participants at three-day, grade-level workshops held throughout the summer. 

This memo provides the first analysis of the effectiveness of the TNCore training on teacher practice 
and student achievement. The findings suggest that attendance at the summer training sessions made a 
significant difference to student achievement and teacher effectiveness, as measured by both observer 
ratings and value-added scores based on statewide TCAP math tests. These results remain consistent even 
as we apply evaluative approaches that use multiple years of teacher data and control for prior scores and 
school effects, suggesting that these results are not biased by participant selection. 


KEY FINDINGS 

• We consistently find positive and significant effects of the TNCore math training on participants' 
instructional practice and on their effectiveness at raising student test scores. These results remain 
consistent using methods that control for previous year scores, school-level inputs, and for the fixed 
characteristics of teachers. 

> Participants' gains on observation scores were equivalent to about half of the gains made by the 
average teacher between the first and second year of teaching. 

> The gains in instructional practice ratings were largest for the practices emphasized in the training 
sessions, including skills such as questioning, providing academic feedback, and teaching problem- 
solving techniques. 

> Participants' gains in effectiveness as measured by the Tennessee Value-Added Assessment System 
(TVAAS) translate into the equivalent of approximately one extra week of learning for each of their 
students than we would have expected had they not attended the training sessions. 

• Participants who had a Core Coach working at their school made significantly greater estimated 
increases in questioning practices compared to participants without this support. 

• For Core Coaches, we find some evidence that the coaching process was associated with 
improvements in their own classroom teaching; however, we are uncertain whether these 
improvements can be attributed to their role as coaches. 
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INTRODUCTION 


O ver the next several years, 45 of 50 states and the District of Columbia are planning to transition to a 
new set of instructional standards designed to guide teaching in each grade and subject. The Common 
Core State Standards (CCSS) are meant to focus on the skills that students will need to be ready for 
college and career and help our children and our country stay competitive. However, there is considerable 
evidence that many teachers across the country feel unprepared to take on the increased demands of 
these new learning standards.^ For instance, a national survey for the American Federation of Teachers 
conducted in March 2013 found that 44 percent of teachers said their districts were "somewhat prepared" 
or "not prepared" to implement the standards. Another survey, conducted across 45 states, found that 
less than one-fifth of math teachers in the sample had received more than 20 hours of professional 
development related to the CCSS.^ 

The Tennessee Department of Education (TDOE) has taken a unique approach to CCSS 
implementation by sponsoring intensive statewide professional development (TNCore 
training) designed to improve student achievement by easing the transition to the 
new standards. In the spring of 2012, through a partnership between the department 
and the University of Pittsburgh's Institute for Learning, the Tennessee Department of 
Education selected and trained about 200 current Tennessee math teachers to serve as 
"Core Coaches," who would prepare their peers to teach the new Common Core State 
Standards. The content of the training focused on research-based instructional shifts 
that would increase student achievement in mathematics. That summer, thousands 
of third through eighth grade math teachers participated in the TNCore math training 
sessions led by the first round of state Core Coaches. In 2013, the state intensified its 
commitment to the peer training, selecting and training nearly 700 math and literacy 
Core Coaches to lead trainings for more than 30,000 educators across the state. 

This memo offers the first rigorous evaluation of the state-led efforts at TNCore training. We focus on two 
central research questions: 

• Did serving as a Core Coach lead to better classroom practice and/or greater effectiveness at raising 
their own students' achievement? 

• Did attending the coach-led TNCore math training lead to better classroom practice and/or greater 
effectiveness at raising student achievement? 

We begin with simple comparisons of outcomes between Core Coaches, training participants, and non- 
participants, moving to a series of more complex models to test the validity of the initial conclusions. 
Across models, we consistently find positive and significant effects of the sessions led by Core Coaches on 
the participants' observed instructional practice and on their effectiveness at raising student test scores. 
For coaches, we find some evidence that the coaching process was associated with improvements in their 
own classroom teaching; however we are uncertain whether these improvements can be attributed to 
their role as coaches. 


Many teachers 
acrass the cauntry 
feel unprepared 
ta take on the 
increased demands 
of these new 
learning standards. 


1 Bill and Melinda Gates Foundation. (2012). Primary sources: American's teachers on the teaching profession. New 
York: Scholastic, inc. 

2 Davis, J., Choppin, J., McDuffie, A. R., & Drake C. (2013). Common Core State Standards for Mathematics: Middle 
school mathematics teachers' perceptions. University of Rochester: Warner Center for Professional Development 
and Education. 
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BACKGROUND 


Tennessee Core Coach Training Model 

T he Tennessee Core Coach training model was designed to develop a network of teachers with a deep 
content and pedagogical knowledge of the Common Core Content Standards for Mathematics and 
Standards for Mathematical Practice who could then pass the knowledge on to their peers. Coaches were 
Tennessee teachers selected via a competitive application and interview process. Coaches received eight 
days of intensive grade-level training provided by the Institute for Learning. By engaging first as learners, 
coaches had opportunities to deepen their own content and pedagogical knowledge, as well as their 
understanding of the ways in which adults learn. During the training, coaches engaged in math lessons 
and reflected on the content learned, how it was learned, and what supported that learning. They also 
analyzed teaching and learning through the lens of pedagogical and content tools. Coaches then delivered 
training to participants at three-day, grade-level workshops held throughout the summer. As part of the 
model, principal Core Coaches were also selected and trained to lead one day of training specifically for 
school leaders attending the three-day training. 

The training given to both coaches and participants was designed to challenge 
teachers' assumptions using a transformative approach. Transformation focuses on 
designing learning experiences that challenge participants' current thinking and often 
startle them into new beliefs.^ Within the training, the goal was for teachers to engage 
directly with student work and assess the strength of their current teaching methods 
while beginning the process of constructing new and more powerful practices. 

During the coach-led training sessions, coaches helped their peers better understand 
new focus areas for grades 3-8 math by working through high level constructed 
response assessment tasks and facilitating model lessons using an instructional 
task. Participants were asked which content and practice standards were addressed 
through the tasks and lessons. They discussed their ideas to deepen their 
understanding of the meaning and expectations of the standards. 

The method of using peer coaches to deliver the training has several potential advantages. The coaches gain 
lasting knowledge that should positively influence their own teaching and students' learning. The coaches 
then pass that knowledge to participants to implement in their own practice. In addition, the coaches are 
available at the school and district level to share their expertise with other teachers to broaden the impact 
of the training. TDOE and the IFL also made all materials used in the training available online to educators. 
These materials included constructed response assessment tasks, instructional tasks, a task analysis guide 
used to determine the cognitive demand of written tasks, and examples of lesson structure and routines. 

It is worth noting that this model of using district personnel as teacher 
trainers makes it more difficult to evaluate the results of the training. 

Upon returning to their districts. Core Coaches and participants were 
often asked to provide training to and share materials with teachers 
within their districts and schools who did not attend the training (see 
Figure 1). In this case, we would detect less of a difference in outcomes 
between those who did and did not attend training, even if the trainings 
actually did have a positive effect since our data cannot entirely 
distinguish between training participants and non-participants. This 
phenomenon suggests that our estimated results might be lower than 
the actual impact of the TNCore training. 


Figure 1. Who received the training? 



By engaging first as 
learners, caaches had 
opportunities to deepen 
their own content 
and pedagogical 
knowledge, as well as their 
understanding of the ways 
in which adults learn. 


3 Kaser, J., Mundry, S., Stiles, K., & Loucks-Horsley, S. (2002). Leading ever day: 124 actions for effective leadership. 
Thousand Oaks, CA: Corwin Press. 
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DATA 

Common Core Math Coaches, Training Participants, and Non Participants 

T he 194 Common Core math coaches in 2011-12 represented 163 schools in 56 districts. The dark blue 
in Figure 2 indicates the counties where coaches were located, and the light blue indicates counties 
that had participants but no coaches. Only one district had no Core Coaches or training participants. Most 
Core Coaches were classroom teachers, although some held other roles in schools and districts, such as 
instructional coaches or administrators. In this study, we focus on Core Coaches who were classroom teachers. 

While teachers from private schools, school and district math coaches and administrators, and university 


Figure 2. Tennessee Counties with 2012 Math Common Core Coaches 



instructors attended the training, the focus of this study is on participants who are public school classroom 
math teachers. About 6,000 participants had an observation score from 2012 or 2013, meaning they were 
a public school teacher. These participants represented almost 1,700 schools across 135 districts. About 
11 percent of the participants had a coach located in their school, and about 5 percent of participants had 
a principal who served as a principal Core Coach. The majority of participants (77 percent) had a school 
leader who also attended the training. 

Non-participants are math teachers in grades 3-8 who did not attend the training. Teachers were identified 
as non-participants if they had TVAAS math estimates for grades 4-8 from 2011-12 school year or grades 3-8 
from the 2012-13 school year but were not on the participant list. Third grade teachers did not have a TVAAS 
estimates in 2011-12. 


Outcome Measures 

We focus on two central measures to identify the effects of the 2012 TNCore math training on teaching and 
learning, teacher observation ratings and a measure of teacher contribution to student achievement. These 
measures are available as part of the statewide educator evaluation system first implemented in the 2011-12 
school year. While these represent our best set of current measures, it is important to note that student 
achievement is measured using Tennessee Comprehensive Assessment Program (TCAP), and while the TCAP 
tests have been focused to more closely align to the CCSS, they are not yet perfectly aligned (see Figure 3). 
Thus, it is possible that, as a result of the training, teachers might become better at teaching the skills that 
will be tested by the next iteration of Tennessee tests but that are not picked up by the current tests. 

Classroom Instructional Practices, in Tennessee, observations of classroom instructional practices 
count as 50 percent of the statewide educator evaluation system. Teachers are observed multiple times 
each year using a state-approved observation rubric. Most districts (comprising about 80 percent of teachers 
statewide) have chosen to use the state observation rubric that is broken into four domains: instruction, 
environment, professionalism, and planning. Each domain contains multiple indicators, scored 1 through 
5 during classroom observations with 5 being the highest possible rating. For the purposes of this study, we 



Figure 3. 

Challenges to Evaluating Professional Development 


There are several significant challenges to evaluating the effects 
of professional development (PD) on teaching and learning. 
While an ideal evaluation would compare a series of long-term 
outcomes between two equivalent groups of teachers, one of 
which had been randomly assigned to receive a certain set of 
trainings and one of which had gone without this treatment, PD 
rarely takes place in this way. As a result, any evaluation faces 
several threats to its overall validity. 

First, the outcome measures may not be aligned with the target 
or focus of the PD. For example, if PD is centered on deepening 
questioning practices, but the outcome measure evaluates 
questioning practices based on frequency rather than depth, we 
may conclude that the PD has not been effective even though 
teachers are truly changing their practice. 

Second, it is difficult to establish an appropriate comparison group. 
Teachers who voluntarily attend PD sessions may start at a higher level 
or have a greater propensity to improve than those who choose not to 
attend prior to the training itself Moreover, teachers who did 


not attend the training may still receive aspects of the training 
through other routes. 

Third, teachers may attend multiple training opportunities or 
may receive other supports that impact their practice, making it 
difficult to isolate the effects of a single PD experience. 

Finally, changes in teacher practice may take time, meaning that 
immediate impacts are difficult to detect. This memo uses the 
following questions to guide our evaluation of the TNCore math 
training. 

Questions to ask when evaluating PD: 

1. Are the measures used to evaluate the PD aligned with the 
PD goals? 

2. Who is the appropriate comparison group? 

3. What other supports do teachers receive that make it 
difficult to determine the impact of PD? 

4. Have teachers had the time to integrate the PD into their 
practice? 


focus on the Instruction domain and on the four Instruction indicators that are most aligned with teaching 
practices emphasized in the training: Questioning, Academic Feedback, Thinking, and Problem Solving. Table 
1 provides the average observation scores for 3-8 math teachers in 2012-13.'* 

Teacher Effectiveness at Increasing Student Achievement. The Tennessee Value-Added 
Assessment System (TVAAS) measures the impact schools and teachers have on their students' academic 
progress. TVAAS, which is based on student results from the Tennessee Comprehensive Assessment 
Program (TCAP), controls for students' previous achievement levels to isolate growth during a particular 
classroom year. For teachers In tested grades and subjects, a 1 to 5 composite TVAAS score counts as 35 
percent of the teacher evaluation model. TVAAS scores for grades 3-8 are calculated at the subject and 
grade level and measure progress using 2009 as the base year comparison. Table 1 also shows the average 
TVAAS score in standard deviation units.** 


Table 1. Average 2012-13 Classroom 
Observation Ratings and TVAAS Scores 


Standard 

Mean Deviation Range 

Instruction Domain 

3.89 

0.59 

1 to 5 

Questioning 

3.74 

0.71 

1 to 5 

Thinking 

3.70 

0.76 

1 to 5 

Problem Solving 

3.69 

0.82 

1 to 5 

Academic Feedback 

3.82 

0.71 

1 to 5 

TVAAS Level 

3.74 

1.41 

ItoS 

TVAAS in SD units 

0.70 

1.00 

-4.1 to 4.4 


4 2012-13 classroom instructional practice means only include teachers who also have 2011-12 practice scores. 

5 2012-13 TVAAS means only include teachers who also have 2011-12 TVAAS scores. 
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FINDINGS 


F igures 4 and 5 present the 2012-13 observation and TVAAS scores of coaches, participants, and grades 
3-8 math teachers who did not participate in the training.® As the figures show, coaches scored 
significantly higher on teacher observation and TVAAS than training participants who, in turn, scored 
higher than non-participants, in 2012-13 after attending the TNCore math training. The problem with 
just examining outcomes in the year following the training is that it does not account for the differences 
between coaches, participants, and non-participants that were already in place before the training. As 
figures 4 and 5 show, observation and TVAAS scores were also higher for coaches prior to the training, 
which is not surprising considering that these factors were part of the coach selection process. Training 
participants had significantly higher initial average observation scores than non-participants but similar 
initial TVAAS scores, suggesting either that school leaders selected teachers to attend the training who 
they perceived as better teachers or those teachers self-selected into the training. 

To get at the impact of the training, we need to take on this comparison group problem by finding a way to 
control for the set of initial differences across each training group that may have contributed to changes in 
outcomes (see Figure 3). 


Figure 4. 2011-12 Observation Ratings and TVAAS Scores Before Training 


4.63 



Observation TVAAS 


■ Coaches ■ Participants ■ Non-participants 


Figure 5. 2012-13 Observation Ratings and TVAAS Scores After Training^ 



Method 1: Controlling for Past Performance 

To evaluate the impact of the 2012 TNCore math training on coaches' and participants' observation and 
TVAAS scores, we need to take into account the different starting points of coaches, participants, and non- 
participants in our study. Thus, we use regression analysis to estimate the impact of training on 2012-13 
scores controlling for 2011-12 scores. Regression analysis is a method for estimating the relationship among 
different factors. Here, it provides an estimate of how much outcomes change if a teacher either served as a 
Core Coach or participated in the training. Regression analysis also allows other factors to be held constant 
(i.e., controlled for). This means we can compare scores between coaches, participants, and non-participants 


6 For each of the outcomes, only teachers who had both years of data are included in the averages. More data is 
available across both years on classroom practices than teacher effectiveness because all teachers, regardless of 
grade or subject or number of students taught, receive observation scores. 

7 In general, TVAAS scores were lower in 2012-13 than 2011-12 due to smaller gains in student test scores. 
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in 2013 while holding constant the influence of initial 2012 effectiveness ratings. We also controlled for the 
teacher's grade and experience level.® 

Holding 2011-12 observation scores constant, we find that being a coach or a participating in the training 
was associated with statistically significant increases in all of the classroom instructional practices examined. 
The gains were at least twice as large for coaches as for participants. This is not surprising given that coaches 
received more intensive exposure to the training via the eight days of training and delivering the content 
in the summer sessions. Interestingly, the relationship appears even larger when we look at the individual 
indicators rather than the instructional domain overall. This suggests that the training might have been 
successful in leading teachers to implement the teaching practices most emphasized in the training. 


On average, this method of analysis suggests that serving as a Core Coach was associated with a third of a 
point increase on observation scores for teachers in practices for teaching problem solving and around a 
quarter of a point increase on practices around academic feedback, student thinking, and questioning (see 
Figure 6). The magnitude of the relationship was smaller, but still statistically significant, for participants. 

Participants gained between one-tenth and one- 

twentieth of a point across the four instructional Figure 6. Estimated Difference in 2012-13 Observation Ratings for 

indicators and overall instructional domain. Coaches and Participants Compared to Non-Participants, Controlling for 

Past Performance 


When we use the same method to look at effects 
on teachers' value-added scores, we similarly find 
that coaching or participating in the training was 
associated with statistically significant increases 
in 2012-13 TVAAS scores.® Serving as a coach 
was associated with about one-fifth of a standard 
deviation increase in teacher effectiveness and 
participating in the training was associated with 
about one-twelfth of a standard deviation increase in 
effectiveness (see Figure 7). More details about the 
models and results are located in the Appendix. 

While these findings provide some evidence that 
perhaps TNCore math training did have meaningful 
effects on teaching practice and students' 
achievement, this method of adding controls for 
teachers' starting points is not sufficient for isolating 
the impact of the training. Coaches, participants, 
and non-participants do not necessarily share the 
same teaching supports (see Figure 3). For example, 
coaches and participants might tend to come from 
schools that are highly invested in the CCSS and 
that have deployed a host of resources toward the 
implementation of the new standards, whereas 
non-participants might instead be drawn from the 
schools that are focused on other issues. Our next 
method builds upon method one by continuing to 
control for initial starting point, but also taking these 
additional concerns into account. 



Figure 7. Estimated Difference in TVAAS Scores for Coaches and 
Participants Compared to Non-Participants, Controlling for Past Performance 



8 A beginning teacher is defined as having only one or two years of experience (being in their second or third year 
of teaching). First year teachers were not included in the study because they do not have prior scores. 

9 TVAAS scores are calculated at the grade level so teachers who teach multiple grade levels receive estimates 
for each grade level. Results could be biased if teachers who teach multiple grades are either more or less 
likely to attend training and more or less likely to be effective. Therefore, we weighted the analysis to account 
for this variation. 


Figure 8. 


What do improvements in classroom instructional practices mean tor students? 


• Students received more frequent, high-quality feedback. 

• Students received instruction in different types of thinking, 
such as analytical, creative, and research-based thinking. 
Students also received more opportunities to analyze problems 
from multiple perspectives. 


• Students were asked questions that advanced their 
understanding and asked to cite evidence for their reasoning. 

• Students engaged in activities requiring them to use a variety 
of problem-solving strategies, including predicting outcomes, 
generating ideas, and improving solutions. 


Method 2: Taking Environnnenf info Account 

The following results continue to control for grade level, teachers' experience level and their previous scores 
on observation and TVAAS, but they add an additional constraint in that all comparisons occur only between 
coaches, participants, and non-participants who taught within the same school. Thus, the effects of the 
training are measured only in schools that had both non-participating math teachers and participating math 
teachers or coaches. This model, known as a school fixed effects model, controls for factors that are the same 
within schools, such as having the same administrative staff conducting observations, having the same school 
orientation toward the Common Core State Standards, and teaching similar student populations. 

Results from the school fixed effect models are quite similar to results of the basic models presented above, 
strengthening the case that these results actually represent the impact of the TNCore math training. 

In the case of teacher observation ratings, results are generally unchanged when including school fixed 
effects. Figure 9 demonstrates the magnitude of the estimated gains in classroom observation ratings for 
teachers who either served as Common Core Coaches or participated in training sessions as compared with 
those in the same school who did not attend the training. 

Estimates of the program's impact on teacher value-added also look quite similar with the addition of 
school fixed effects. Again, being a coach was associated with about a fifth of a standard deviation increase 
in teacher effectiveness, while participating in the training was associated with smaller but still statistically 
significant increases (see Figure 10). 

These results add confidence that the TNCore math training was significantly associated with improvements 
in teacher quality. A final concern, however, is that the difference between coaches, participants, and 
non-participants lies not in their starting point or in their environment but rather in their likelihood of 
improvement overtime. In other words, coaches and participants might represent the group of teachers 
who are more likely to achieve these increases without training, while non-participants might have improved 
less even if they had attended the training. 


Figure 9. Estimated Difference in Observation Ratings for Coaches 
and Participants Compared to Non-Participants, Controlling For 
Environment and Past Performance 



Figure 10. Estimated Difference in TVAAS Scores for Coaches and 
Participants Compared Non-Participants, Controlling For Environment 
and Past Performance 
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Method 3: Adjusting for Teacher Characteristics 

In both previous methods, our comparison group was the group of teachers who had not participated 
in TNCore math training, either as coaches or participants. However, these methods did not allow us to 
consider the possibility that coaches and participants might have moved on completely different paths 
toward teaching improvement than non-participants regardless of the specific training they attended. 

Here, we shift course and compare individuals to their own selves in the years before they participated in 
training. This model, known as an individual fixed effect model, holds constant anything within teachers that 
does not change over time, thereby controlling for unobservable characteristics such as their likelihood for 
self-improvement. We use TVAAS data dating back to the 2007-08 school year. Using multiple prior years of 
TVAAS performance allows us to examine whether the teachers improved more during the year following 
participation in the training than we would have expected given their previous scores. “ Since statewide 
teacher evaluation only began in 2011-12 in Tennessee, we are unable to employ this method for teachers' 
observation ratings. 


Training participants impraved significantly more in 2012-13 than we would have 
expected even knowing their TVAAS scores from the previous several years. 


Interestingly, these results are different than those in previous sections. Serving as a coach is no longer 
significantly associated with an increase in teacher effectiveness scores (see Figure 11). While our previous 
estimates showed that coaches' gains on TVAAS outpaced participants and non-participants, these later 
results suggest that the gains might not have been caused by the coaching experience. Coaches were already 
on a trajectory to outperform participants and non-participants, even before the training took place. 

At the same time, the estimates of the impact of the training on TVAAS scores for training participants are 
nearly identical to the results we reported in previous sections. Again, being a participant as associated with 
an increase of about 0.08 standard deviations in teacher effectiveness. In other words, training participants 
improved significantly more in 2012-13 than we would have expected even knowing their TVAAS scores from 
the previous several years. These matching findings across models provide evidence that the effect we are 
detecting can be ascribed to the TNCore math training that the participants attended in summer 2012. 


Figure 11. Estimated Gains in TVAAS Scores for Coaches 
and Participants, Adjusting for Teacher Characteristics 


Coach 


not 

significant 


Participant 


0.1 


0.2 


0.3 


TVAAS Performance 
(standard deviation units) 


10 The model additionally includes controls for year and grade level to account for differences in aggregate trends 
overtime by year and grade level. 
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INTERPRETING THE RESULTS 


Each method described above uses a different technique to address the challenges of evaluating professional 
development. So what can we conclude when we look across models? Coaches experienced gains in 
observation scores and in student growth but they appear to be a select group who may have made greater 
gains than their peers in 2012-13 with or without their participation in coaching sessions. 

For those who participated in the training, we find a consistent and significant effect of attending the 
training on both their instruction and classroom impact. Participants score about one-tenth of a point 
higher on the 1-5 observation scale for various observation indicators, and they rate about 0.08 standard 
deviations higher on TVAAS scores than we would have predicted absent the training. 

Now that we have evidence that the training had a positive impact, it is helpful to think about the 
implications for students. One way to think about the impact of the training is by comparing the average 
effectiveness of coaches to non-coaches. On average, coaches are about one-third of a standard deviation 
higher in teacher effectiveness than non-coaches (i.e., participants and non-participants). After attending the 
training, participants are estimated to improve their effectiveness at increasing student achievement by 0.08 
or one-twelfth of a standard deviation. This means participants would be about 25 percent closer to coaches' 
estimated effectiveness than they were before the training. Figures 8 and 12 also offer help interpreting the 
magnitude of these results in terms that offer more insight into the impact of the effect. 

In the terms described in Figure 12— about half of the difference between a first- and second-year teacher as 
measured by observation ratings, and one week of student achievement as measured by TVAAS— the gains 
that participants made do not seem particularly large. At the same time, these represent results from a single 
three-day training session that participants attended during the summer months, and they contrast with 
several other rigorous studies of recent professional development initiatives that have found no significant 
effects on teaching or learning. Moreover, the 2012 training evaluated here represented the first time that 
the state attempted summer training sessions at such scale, with the subsequent, larger training in 2013 
benefiting from several lessons learned during the previous summer. 


11 Yoon, K, S., T, Lee, S. W. Y., Scarloss, B., & Shapley, K. L. (2007). Reviewing the evidence on how teacher 

professionai development affects student achievement. National Center for Educational Evaluation and Regional 
Assistance, Institute of Education Sciences, U.S. Department of Education 


Figure 12. 

How large are the effects? 

Classroom Instructional Practices. On the 1-5 observation 
scale. Core Coaches score about one-third of a point higher and 
participants score about one-tenth of a point higher compared 
to non-participants. While these gains may seem quite small, it is 
important to note that observation scores for Tennessee teachers 
tend to range between three and five, meaning that a quarter- 
point gain can be quite meaningful. Another way of thinking 
about the gain is by looking at average difference across teacher 
cohorts. The one-third of a point predicted change in coaches' 
problem solving scores is about the same size as the average 
increase on probiem solving for teachers in their second year 
compared to their probiem soiving scores in their first year. 
The magnitude of the impact was smaller for participants. For 
example, participants are associated with a 0.08 Increase In 
academic feedback. This is about half the size of the average 
increase in academic feedback for second year teachers. 


Teacher Effectiveness. We draw on learning time calculabons 
used by The Education Trust to interpret our findings. Research 
examining student achievement on a nationally norm-referenced 
test reported expected Increases, on average, by about 0.42 
standard deviations in math over the course of a calendar year. 
Research conducted by SAS Institute, Inc. found that in Tennessee 
a one standard deviation (SD) increase in teacher effectiveness 
results in a 0.15 to 0.22 SD increase in student achievement. Using 
the lower estimate, a one SD increase in teacher effectiveness 
is associated with an increase in student outcomes of about 
0.15 standard deviations, which equates to about 3.5 months of 
learning in math. Using this interpretation, being a core coach 
was associated with student achievement gains of about three 
weeks and participating in the training was associated with 
student achievement gains for about one week. If the higher 
estimate of the relationship between teacher effectiveness and 
student achievement was used, the estimated impact would be 
4.2 weeks of student achievement for students of Core Coaches 
and 1.6 weeks of student achievement for students of participants. 


• • 


• • 
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Importantly, the calculations about the effects of the training on student achievement also allow us to 
provide a rough cost-benefit calculation using data from other research studies about the economic impact 
of higher student test scores. While the cost-benefit projections require several assumptions, they also offer 
a useful gauge as to whether the state is spending its Race to the Top dollars wisely. 

The total cost of the 2012 summer math training was about $3.2 million. This included materials, training 
sites costs, lodging for trainers, and payments to Core Coaches for leading the training sessions. We estimate 
that the training was provided to about 6,000 public school teachers in the state of Tennessee. Thus, the 
cost averages to about $500 per teacher.^ Hanushek (2004) calculates that a one standard deviation 
increase in student achievement translates into about $60,000 increase in lifetime earnings. We found that 
participating in the training was associated with 0.08 standard deviation increase in teacher effectiveness 
or a 0.01 increase in student achievement. Therefore, students of teachers who participated in the training 
would be expected to experience about a $600 lifetime increase in earnings. If the 6,000 teachers who 
attended the training taught about 30 students each, this would translate into about 180,000 students 
earning an additional $600 over their lifetime or $108 million. This estimated benefit is based on a single 
year of improved teacher effectiveness. We expect that the improvements attributed to the training would 
continue to benefit students of teachers who attended the training in future years. While these calculations 
provide a useful method of estimating the financial benefit of the training, they are just estimates and should 
be interpreted with caution. 


Impact for Participants with Access to Coaches at their School 

It is also useful to think about whether there were particular attendees who benefited most from the 
training. Because Common Core Coaches were also Tennessee teachers (and principals), the participants who 
had coaches actually working in their schools likely had greater access to ongoing feedback and support for 
implementing the principles of the training than those who only attending the three-day summer training. 
This section addresses the question: Did participants in coach schools benefit from the training more than 
participants in non-coach schools? 

Since having access to a Core Coach would be the same for all participants in the same school, we are 
unable to use a school fixed effects approach comparing teachers in the same school. Therefore, we used 
a district fixed effects specification that compares teachers in the same district to each other, accounting 
for differences in supports that are similar across districts. We compared participants with Core Coaches in 
their schools to participants without Core Coaches in their schools. 

Participants with a Core Coach at their school had significantly greater estimated increases in questioning 
compared to participants without this support. Participants with coaches in their schools were predicted to 
improve questioning practices by 0.08 compared to participants without access to coaches. The differences 
in predicted increases were not statistically significant between the two participant groups for the other 
instructional practices. Thus, we conclude that having access to a Core Coach was associated with greater 
improvements in questioning practices for training participants. We conducted a similar analysis to look at 
the impact for participants having access to a coach on teacher effectiveness outcomes. However, due to 
the smaller sample size of teachers with TVAAS scores from both years, the groups of participants with and 
without coaches at their schools were too small to detect differences. 


12 While the 2012 summer training actually served about 13,000 participants (including school and district leaders 
and other personnel), we are most interested in the impact of the training on classroom teachers' instructional 
practices and effecbveness at increasing student achievement. Therefore, we calculate the cost of training for 
just Tennessee public school classroom teachers. 

13 Hanushek, E. A. (2004). Some simple analybcs of school quality (No. wl0229). National Bureau of Economic Research. 
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CONCLUSIONS 


T his memo reports on the results of a study of the impact of Tennessee Department of Education led 
training sessions for grades 3-8 math teachers across the state in the summer of 2012. Using a variety 
of analysis techniques, we consistently found a statistically significant relationship for attending the three- 
day training session on teachers' instructional practices in the areas of questioning, feedback, problem 
solving, and thinking, and on teachers' contributions to student achievement. Though the magnitude 
findings may appear small for an individual teacher, when compounded across the 6,000 participants who 
each teach numerous students, the training seems to have had a great benefit for students in Tennessee. 

While we have taken a number of steps to identify the true impact of the training, it is important to 
remember that the challenges of evaluating professional development make this task difficult. Due to the 
nature of the training model, we cannot be certain whether non-participants are the right comparison 
group as some non-participants may have received access to the training principles. If some non- 
participants received the training and their practices improved as a result, this would bias the estimate of 
the impact of the training on coaches and participants downward, meaning that the impact could be larger 
than we are estimating. Additionally, we may see a greater relationship for attending the training on teacher 
effectiveness if student achievement was measured with an assessment that was more closely aligned with 
the Common Core State Standards. 

Lastly, the training was designed to shift teachers' thinking about teaching and often teachers need time for 
this shift to take place and to integrate the lessons learned from the PD into their practice. Our investigation 
focused on only the initial year after the training. However, it may actually take longer than a year for 
teachers to change their practice and for those changes to result in increases in student achievement. 
Therefore, it is important to continue to investigate the long-term impacts of the training. 


Click here to view the Technical Appendix. 


If you have questions or comments regarding this report, please email Education. ResearchjStn. gov . 


